Skip to content

feat: pre-warm tailscale path to k3s server on game node start#362

Open
Flegma wants to merge 1 commit into
mainfrom
feat/agent-node-tailscale-prewarm
Open

feat: pre-warm tailscale path to k3s server on game node start#362
Flegma wants to merge 1 commit into
mainfrom
feat/agent-node-tailscale-prewarm

Conversation

@Flegma

@Flegma Flegma commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Problem

GET /game-server-node/script/:id (game-server-node.controller.ts) generates the curl | bash script the panel gives you to add a game/GPU node. It provisions the node as a k3s agent joining the control plane over Tailscale.

After a reboot, the Tailscale direct path to the k3s server can come up one-way (tx>0, rx 0). tailscaled self-reports healthy (BackendState=Running, empty Health), so the existing 5stack-tailscale-state-check passes every cycle, yet the agent cannot reach the control plane. k3s-agent then loops on failed to get CA certs ... context deadline exceeded, never joins the cluster, no pods (including game-server-node-connector) schedule, and the node shows Offline in the panel.

Observed live: a GPU node stuck ~27 min after a reboot; a manual tailscale ping <server> re-established the path and it recovered on its own within one retry.

Fix

Add a best-effort ExecStartPre drop-in (tailscale-prewarm.conf, in k3s-agent.service.d) that runs before k3s-agent starts: if K3S_URL is set, it does a bounded tailscale ping to the server, forcing Tailscale's NAT traversal to (re)establish the direct path bidirectionally, so k3s-agent never validates against a dead one-way path.

  • Prefixed with - and wrapped in || true so a failed or hung pre-warm can never block k3s-agent from starting (the ping is bounded by timeout 15).
  • Does not touch the existing tailscale-state-check logic or the other drop-ins.

Relationship to 5stack-panel#573

This is the canonical agent-provisioning path, so it is the one that protects all future nodes added via the panel. 5stack-panel#573 adds the same pre-warm to the separate game-node-server-setup.sh (standalone-server setup, where it is a no-op).

Testing

  • Installed the identical drop-in on a live agent node and verified end to end through a transient systemd unit: K3S_URL=https://100.90.141.113:6443 parses to 100.90.141.113, the ping ponged, and it is non-blocking.
  • Confirmed the generated shell renders correctly from the JS template literal (escaped \${...} becomes ${...}, bare $H preserved); tsc clean and backtick balance even.

Note

Guards the reboot case (path warmed at startup). It does not add steady-state one-way-path detection to the periodic tailscale-state-check, which still only inspects tailscaled's self-reported health; that would be a follow-up.

@Flegma
feat: pre-warm tailscale path to k3s server on game node start
fcc3b3d 
The script served by GET /game-server-node/script/:id is the canonical
agent-provisioning path (curl | bash from the panel). After a reboot the
Tailscale direct path to the k3s server can come up one-way (tx>0, rx0);
tailscaled self-reports healthy, so the existing tailscale-state-check
passes while the agent cannot reach the control plane: k3s-agent hangs
validating the connection, no pods schedule (including
game-server-node-connector), and the node shows Offline.

Add a best-effort ExecStartPre drop-in (tailscale-prewarm.conf) that,
before k3s-agent starts, runs a bounded tailscale ping to the server
parsed from K3S_URL, forcing the direct path to establish
bidirectionally. Prefixed with - and wrapped in || true so a failed
pre-warm can never block k3s-agent from starting. Mirrors 5stack-panel
PR #573.
@Flegma
Flegma force-pushed the feat/agent-node-tailscale-prewarm branch from 0f50acd to fcc3b3d Compare July 20, 2026 19:30
@Flegma

Flegma commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Code review

Found 1 issue, now fixed in fcc3b3d.

systemd %-specifier collapse in the host parse. The original ${H%%[:/]*} runs inside a systemd ExecStartPre=, where systemd collapses %% into a single literal % before bash ever sees it, even inside the single-quoted bash -c. Confirmed on the target's systemd 255 with a transient unit: H=a:b:c produced a:b (bash shortest-match %), not the intended a (longest-match %%). It still yields the correct host for the real K3S_URL shape (https://<ipv4>:6443, a single :, where shortest and longest match coincide), so it was not a live failure, but the operator did not do what it read as and would mis-truncate a multi-separator host.

Fixed by parsing the host with a %-free pipeline (cut -d: -f1 | cut -d/ -f1). Verified through a transient systemd unit: a:b:c to a, host/p:9 to host, https://<ipv4>:6443 to <ipv4>, empty to empty. The identical drop-in is installed and verified on a live agent node, and k3s-agent was not disturbed.

api/src/game-server-node/game-server-node.controller.ts

Lines 717 to 726 in fcc3b3d

cat <<-'DROPIN' >/etc/systemd/system/k3s-agent.service.d/tailscale-prewarm.conf
[Unit]
After=tailscaled.service
Wants=tailscaled.service
[Service]
ExecStartPre=-/bin/bash -c '. /etc/systemd/system/k3s-agent.service.env 2>/dev/null || true; H=$(echo "\${K3S_URL#*://}" | cut -d: -f1 | cut -d/ -f1); [ -n "$H" ] && echo "[5stack] pre-warming tailscale path to $H" && timeout 15 tailscale ping -c 3 "$H" >/dev/null 2>&1 || true'
DROPIN

@Flegma Flegma mentioned this pull request Jul 20, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Flegma